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Executive summary 



C ommunity colleges are designed to 
be open-door institutions, and they 
enroll a much wider variety of 
students than baccalaureate - 
granting colleges. Community 
colleges have always played a crucial role in 
providing access to college. During the last 
decade, however, educators and policy-makers 
have shifted their attention 
also to the success of 
students once they enter 
community college. As a 
result, accreditation 
agencies and state regulators 
are increasingly scrutinizing 
measures of student 
outcomes such as persis- 
tence and completion rates. 

At the same time, 
national initiatives by 
foundations and the U.S. Department of Education 
are focused on developing policy and institutional 
practices that will improve success rates for 
community college students. This report has been 
written as part of one of those initiatives. In 2003, 
Lumina Foundation for Education joined eight 



other organizations to launch Achieving the 
Dream: Community Colleges Count. Achieving 
the Dream is based on the premise that research 
about and at community colleges must play a 
central role in any strategy to increase student 
success. 

This report presents a critical analysis of the 
state of the research on the effectiveness of four 

types of practices in 
increasing persistence and 
completion at community 
colleges: 1) advising, 
counseling, mentoring and 
orientation programs,- 

2) learning communities,- 

3) developmental educa- 
tion and other services for 
academically under- 
prepared students,- and 

4) college-wide reform. We 
use this analysis to draw substantive lessons about 
effective institutional practices, to identify 
promising areas for future research, to evaluate the 
state of program-effectiveness research at 
community colleges, and to make recommenda- 
tions for improving related research. 



Research about and at 
community colleges must 
play a central role in 
any strategy to increase 
student success. 
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Substantive lessons 

Among the practices and strategies that we 
examined, learning communities appear to have 
the most support grounded in research. In a 
learning community, students go through the 
program as a cohort, and their instruction is 
typically organized around themes. The learning 
community models positive effects on persistence 
and graduation are consistent with the most 
influential theoretical 
perspectives used to study 
retention. Empirical 
research also suggests 
positive effects. Research 
conclusions point out that 
counseling, advising and 
developmental education 
are all crucial for community college students, but 
research has been less helpful in identifying the 
most effective design and organization for these 
services. Major national research projects in these 
areas are now a decade old, and the policy and 
demographic environments have changed 
significantly in the intervening years. Studies of 
broad, college-wide change and the institutional- 
ization of pilot programs are particularly wanting. 
We lack good conceptualizations as well as 
empirical measurement of these types of strategies. 

Improving research 

Research on program effectiveness at commu- 
nity colleges can be improved by addressing 
problems in four areas. First, the large majority of 
the research on program effectiveness in higher 
education is limited to studies of four-year 
colleges. Insights obtained from this research do 
not necessarily translate to effective practices for 
the part-time, working and adult population that 
characterizes community colleges. Second, the 
national data sets that allow comprehensive 
analysis of the experience of postsecondary 
students do not include data on the types of 
specific institutional practices and policies that 
colleges use to increase student success. Third, 



methodological problems thwart definitive 
conclusions about the effectiveness of community 
college policies and practices. Fourth, the 
dissemination and discussion of research on 
community colleges are inadequate. Reports are 
difficult to obtain and usually include too little 
information to allow a judgment about the validity 
of the conclusions. Faculty and even researchers at 
community colleges rarely have the opportunity to 

discuss research on program 
effectiveness in a way that 
allows them to understand 
the strengths, weaknesses 
and lessons of the studies. 

To strengthen our 
ability to choose and assess 
policies and practices that 
will help increase the success of community 
college students, we suggest the following changes 
in community college research: 

Theoretical issues: The dominant theoretical 
perspective on retention and completion, the 
student integration or engagement model, was 
developed based primarily on four-year college 
models with particular emphasis on full-time, 
traditional-aged, residential students. Empirical 
tests of these models have not yielded strong 
support for their application to community 
colleges. Researchers have begun to take into 
account commuter students, but the particular 
characteristics of community colleges and their 
students are still neglected. 

Data availability: Because insufficient national 
data exist on institutional practices, most program- 
effectiveness research is based on samples from 
single institutions. While these can be useful, their 
conclusions are difficult to generalize because 
effects may be based on particular features of the 
college being studied. As much as possible, 
national-level databases, such as those created by 
the National Center for Education Statistics, 
should include programmatic detail to allow 
research on the effectiveness of common practices 



Most program-effectiveness 
research is based on samples 
from single institutions. 
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used to improve student success. Collaborative 
projects within community college districts and 
states, supplemented with increasingly available 
state- or district-level student record data, also 
would be helpful in understanding persistence and 
student success and in overcoming some of the 
limitations of single-campus studies. Even with 
single-institution studies, research that makes use 
of transcript-based longitudinal data can provide 
many important insights. 

Empirical issues: There are some excellent 
studies on program effectiveness at community 
colleges,- in general, however, empirical research 
needs to do a better job of exploiting available 
data and of employing quantitative and qualitative 
methodologies for effectiveness research. Too few 
studies that measure a 
relationship between 
community college 
programs and student 
outcomes are designed in 
ways that support conclu- 
sions about whether that 
relationship is a causal one. 

The following suggestions 
can strengthen the reliabil- 
ity and validity of research 
on these issues. 

Empirical research must, as much as 
possible, control for measured student 
background characteristics. Scores on entry 
assessment tests or information on the high 
school academic record are particularly 
important. 

Under some circumstances, statistical 
techniques can account for unmeasured 
characteristics, such as motivation, that 
might influence student outcomes. These 
techniques are not now used in community 
college research. 

Random-assignment designs address many 
of the most difficult methodological 
problems,- thus, their conclusions are 
particularly useful and influential. However, 



because such studies are costly and 
difficult to administer, they are infrequent. 
Finally, every study of a program must 
include a detailed description of the 
characteristics of the program and of the 
process through which students enter that 
program. This information gives readers 
essential background that allows them to 
interpret the research results and judge the 
validity of the conclusions. 

Research at the community college: Although 
university researchers must pay more attention to 
community colleges, research will have a funda- 
mental influence on the colleges only when it 
plays a more prominent role on the campuses 
themselves. Reformers refer to this as developing a 

'culture of evidence" in 
which institutional research 
functions play a more 
prominent role and faculty 
and administrators are 
more fully engaged with 
data and research about the 
success of their students, 
using those data to make 
decisions. We present six 
suggestions for developing 
that culture: 

1 . Colleges must assess the resources and 
skills needed for effective institutional 
research, recognizing that research is an 
investment. As with even the most 
rewarding investment, its payoff emerges 
only over time. 

2. Colleges must recognize that assessments 
of program effectiveness are difficult and 
involve a continuum of activities and 
analyses. 

3. Projects should combine quantitative 
research on student outcomes with 
qualitative research to elicit insights from 
students about those outcomes. 



Research will have a 
fundamental influence on 
the colleges only when it 
plays a more prominent role 
on the campuses. 
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4. Colleges, states and college associations 
must provide more opportunities for faculty 
and administrators to engage in the 
research process and to discuss evidence 
about student outcomes. 

5. Colleges and states must develop more 
systematic methods to publicize and 
disseminate research findings. 

6. Collaboration among institutional 
researchers at different colleges and 
between college-level and state-level 
researchers should be promoted. 

This report argues that much needs to be done 
to strengthen research on community college 



program effectiveness. Too often, research 
provides ambiguous or weak guidance concerning 
many policies and programs designed to improve 
student retention and success. Of course, despite 
these uncertainties, colleges must continue to 
move forward and act on the best available 
knowledge — even if that knowledge is limited or 
open to alternative interpretations. Our overall 
recommendation is that colleges search for the 
best information they can find and monitor 
progress as thoroughly and rigorously as possible. 
The interaction between research and practice 
involves a continuous conversation within and 
among the colleges, and with outside researchers 
and policy-makers, as practitioners try to improve 
their performance in a constantly changing 
environment. 
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Introduction 



C ommunity colleges are designed to 
be open-door institutions. They 
enroll a much wider variety of 
students than baccalaureate - 
granting colleges: Minority 
students, first-generation college students, students 
with lower levels of academic achievement in high 
school, and students from low-income families are 
all significantly overrepresented in community 
colleges when compared with their enrollment in 
baccalaureate-granting institutions. During the 
decades of rapid expansion 
of community colleges, 
these considerations of 
access were paramount, but 
more recently educators, 
policy-makers, researchers 
and foundations have all 
increasingly turned their 
attention to the actual 
experience of students enrolled in these institu- 
tions. This focus has revealed that community 
college students have low persistence and 
completion rates. Of all first-time college students 
who entered a community college in 1995, only 36 
percent earned a certificate, associates or 



bachelors degree within six years. Although many 
students who did not complete degrees may have 
met other personal goals, policy-makers and 
educators judge these rates to be too low (Bailey & 
Leinbach, 2005). Moreover, completion rates for 
African-American, Hispanic, Native American and 
low-income students are lower than the overall 
numbers, indicating inequitable racial and income 
gaps. 1 

National initiatives by foundations and the 
U.S. Department of Education are focused on 

developing policy and 
institutional practices that 
will improve retention, 
completion and other 
measures of success for 
community college 
students. These initiatives 
also aim to reduce the 
achievement gaps between 
students from different racial/ethnic and income 
groups. This report has been written as part of one 
of those initiatives. In 2003, Lumina Foundation 
for Education joined with eight other organiza- 
tions to launch Achieving the Dream: Community 
Colleges Count. 2 In early 2005, Achieving the 



Community college students 
have low persistence and 
completion rates. 
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Dream is working with 27 community colleges in 
five states to help them increase retention, 
completion and success for community college 
students, particularly those in groups that have 
been underserved in higher education, such as low- 
income students, students of color and first- 
generation college students. Research and data are 
at the heart of all aspects of Achieving the Dream. 
First, colleges are expected to use institutionally 
relevant data and research to measure the current 
levels of students' success. Data are to be disaggre- 
gated by race, ethnicity and income (when 
possible) and to be used to diagnose the causes of 
the problems they identify. Moreover, participating 
colleges are expected to select strategies and 
interventions for which there is empirical evidence 
of effectiveness. Finally, the initiative is designed to 
help the colleges strengthen their own capacity to 
conduct, and particularly to use, institutional 
research on an ongoing basis to identify problems 
and choose and assess alternative solutions. This 
broader objective is articulated in the admonition 
to develop a "culture of evidence," rather than 
relying on a "culture of anecdote" in which 
community colleges justify themselves simply by 
telling encouraging stories about individuals who 
overcome daunting barriers to succeed at commu- 
nity colleges. 



The goal of this report is to provide back- 
ground information and analysis to support a 
broad effort to strengthen the use of data and 
research in improving student outcomes at 
community colleges. It starts by presenting an 
overview of the state of research on the effective- 
ness of institutional programs and policies 
designed to improve community college student 
outcomes, particularly persistence and completion. 
The next section discusses the most common 
theoretical perspectives used in the study of 
retention and completion in higher education and 
questions the appropriateness of applying those 
perspectives to community college students. We 
then review research on effectiveness of commu- 
nity college programs in four broad areas: student 
services and advising, developmental education, 
learning communities and institution-wide reform. 
In addition to drawing programmatic lessons from 
this literature, we use these analyses to support 
generalizations about the state of program- 
effectiveness research on community colleges. We 
end with conclusions and recommendations for 
research about community colleges and for reforms 
in research done on community college campuses 
— reforms that can help develop a "culture of 
evidence" at the colleges. 
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The state 
persistence 



of research on 
and completion 



A wealth of research exists on 

persistence and completion in 
higher education. In 1991, Ernest 
Pascarella and Patrick Terenzini 
published an 800-page volume 
reviewing almost 3,000 studies on How College 
Affects Students, including many studies of retention 
and completion. Since then, research on the topic 
has continued in journals and in unpublished 
reports. But this vast 
landscape of papers and 
reports yields relatively few 
concrete insights about our 
specific topic: the effects of 
institutional policies on 
community college retention and completion. 

This dearth of insight results largely from four 
problems. First, most of this research is about four- 
year colleges. Second, available national (or even 
multi -college) data do not have good measures of 
institutional practices designed to promote 
retention and completion. Third, flawed method- 
ology often thwarts efforts to properly assess 



institutional practices. Fourth, the dissemination 
and discussion of research reports on community 
colleges are inadequate. We shall examine each of 
the four problems in turn. 

1. Overemphasis on the four-year college: 
The vast majority of the research on student 
retention and completion is concerned with four- 
year colleges. In concluding their definitive review, 

Pascarella and Terenzini 
(1991) state that their work 
was "based almost exclu- 
sively on samples of 
traditional college students 
who are age 18 to 22, who 
attend four-year institutions full-time, and who live 
on campus" (p. 632). This review is now almost 15 
years old. There is no question that community 
colleges have attracted more attention during the 
last decade, but this new interest is not reflected in 
published research. A review of articles published 
in five mainstream higher education journals 3 
between 1990 and 2003 by Townsend, Donaldson 



Most of this research is 
about four-year colleges. 
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and Wilson (2004) found that only 8 percent of 
the 2,321 articles even mentioned community 
colleges. 

The lack of research on community colleges is 
a particularly serious problem when it comes to the 
study of retention. Much of the research and 
thinking on retention has been based on the 
concepts of student engagement and integration 
with the college. These concepts are likely to be 
most powerful for residential students, who 
represent a small minority of the student popula- 
tion at community colleges, which are primarily 
commuter schools. But what differentiates the 
community college student body is the predomi- 
nance of part-time students: Only 36 percent of 
community college students 
attend full-time, while 71 
percent of four-year college 
students are enrolled on a 
full-time basis. 4 In addition, 
community college students 
tend to be older, are more 
likely to be working, and are 
more likely to interrupt their 
enrollments. Policies 
designed to retain 18 -year- 
old students living in dorms 
are not likely to be as effective for part-time, 
working students and especially for adults with 
families and full-time jobs. 

2. Lack of data on institutional policies: The 
primary source for national data on institutional 
characteristics is the Integrated Postsecondary 
Education Data System (IPEDS) collected by the 
National Center for Education Statistics (NCES). 
IPEDS includes data on more than 1,000 commu- 
nity colleges, but it has little detail on the types of 
institutional practices colleges use to improve 
retention — student services, pedagogic strategies, 
organizational innovations, etc. Therefore, most 
studies of the effects of institutional practices are 
based on student surveys and often involve only a 
single institution. They are of mixed quality and, 
in any case, produce results that are difficult to 



generalize. To the extent that results depend on 
answers provided by students, the studies also fail 
to measure institutional practices directly. In 
contrast, there are much better national data 
available to study the relationship between 
individual characteristics and retention and other 
educational outcomes. This is interesting research, 
but it does not answer our question — how do 
institutional factors and policies influence 
retention and completion? 

3. Methodological problems: Evaluations of 
institutional practices are notoriously difficult 
because of problems with the attribution of 
causality. Most practices that are studied involve 

discrete programs,- some 
students are in those 
programs, and others are 
not. Studies of the 
effectiveness of the 
programs generally consist 
of a comparison between 
those two groups of 
students, but these types of 
comparisons often do not 
provide enough informa- 
tion to make a valid 
judgment. As long as there is some non-random 
process by which students enroll or are chosen for 
such a program, it may be that any differences 
between participants and non-participants result 
from the selection process, not from the program 
itself. Thus, even though a program shows positive 
results, those results may not hold for other groups 
of students. There is no question that causality 
problems can be difficult, but there are techniques 
that can be used to address them. As this report 
will show, much of the research on community 
colleges fails to make use of these techniques. 

4. Inadequate dissemination and discussion: 
Methodological problems are compounded by the 
methods generally used in community colleges to 
disseminate and discuss institutional policy and 
initiatives. Results are often posted on Web sites, 



Community college students 
tend to be older, are more 
likely to be working, and 
are more likely to interrupt 
their enrollments. 
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frequently in the form of PowerPoint presenta- 
tions, or are presented at conferences and 
conventions, such as those held by the American 
Association of Community Colleges or the League 
for Innovation in the Community College. Most 
of these studies, however, are unpublished and 
therefore may not undergo a rigorous vetting and 
quality-control process. Full reports often are 



difficult to track down and rarely provide enough 
methodological detail for the reader to make a 
judgment about the reliability of the results. In 
general, community college institutional research- 
ers and practitioners get little chance to discuss 
research findings in a way that might allow a more 
comprehensive understanding of the results and 
implications of existing research. 
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Conceptual perspectives 
on retention and completion 



I n this section, we discuss in more detail the 
theoretical approaches to studying student 
retention and how they are used for the 
analysis of community colleges. Vincent 
Tintos ( 1975 , 1993 ) student integration model 
forms the conceptual basis of much of the research 
on persistence and gradua- 
tion. It also has some 
attractive implications for 
institutional policy. Tintos 
model is designed to help 
colleges understand why 
students leave, so the 
institutions can design 
activities to better serve students' needs and 
thereby increase retention and graduation rates. 
Tinto states that students' departures from an 
institution "reflect the character of the individual's 
social and intellectual experiences within the 
institution. Specifically, they mirror the degree to 
which those experiences serve to integrate 
individuals into the social and intellectual life of 
the institution" ( 1993 , p. 51 ). This model differen- 



tiates between social integration , which is measured 
by such factors as interaction with faculty and 
participation in extracurricular activities, and 
academic integration , which is usually measured by 
grades or other indications of academic achieve- 
ment. This perspective implies that institutions 

should develop processes 
and activities that foster 
both types of integration 
among college students. 

Although this model 
resonates with educators and 
portrays an attractive 
college environment, 
empirical analysis has been difficult. Research on 
student integration has a profound causality 
problem. Studies of four-year colleges show that 
students who participate in student organizations 
or interact with faculty persist and graduate at 
higher rates. Still, it does not follow that gradua- 
tion rates will increase if every student joins a 
student organization or interacts with faculty. 
Students may interact with faculty because they and 



Research on student 
integration has a profound 
causality problem. 
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the faculty share values and an orientation toward 
academic activity. For example, Pascarella and 
Terenzini (1991) reported the results of a study 
that concluded: 'educational aspirations are more 
likely to influence contact with faculty than 
contact with faculty is to influence educational 
aspirations" (p. 395). This problem is compounded 
when studies are done on single institutions. In 
these studies, students often are asked to fill out 
questionnaires, and their levels of social or 
academic integration are scored based on their 
answers (for example, see Napoli & Wortman, 
1998). But if the students are all at the same 
institution, then presumably they all have access to 
the same services or activities that might promote 
integration. If the students share the same 
environment, then measures of integration are 
measures of individual characteristics, not 
institutional characteristics. 

Thus, the types of studies 
that show a relationship 
between social integration 
and retention do not imply 
that introducing policies to 
promote integration will 
necessarily increase 
retention. 

In contrast to the theoretical emphasis on 
social integration, Bean and Metzner (1985) 
explicitly developed a model of attrition for 
nontraditional students, both at four- and two-year 
institutions. Their contention was that social 
integration would play a much smaller role among 
these students and that outside "environmental" 
variables would be more important. These 
variables included finances, hours of employment, 
outside encouragement and family responsibilities. 
They also suggested that "goal commitment" and 
"intent to leave" were important for nontraditional 
students and that these students are more focused 
on the economic benefits of their education. 

Since Bean and Metzner (1985) place a great 
deal of importance on environmental factors 
outside the colleges control, their approach would 
appear to leave less potential for an institutional 



response. In their model, the two variables that are 
under the control of the colleges are academic 
advising and course availability. Presumably, 
academic advising should be designed to increase 
goal commitment and influence the students intent 
to leave. Course availability is certainly a logical 
determinant of attrition, especially for nontradi- 
tional students, who generally have a more 
instrumental view of their college education. 

More recently, Braxton, Hirschy and 
McClendon (2004) have attempted to extend 
Tinto's model to "commuter universities and 
colleges." Unlike residential colleges, the authors 
say, "commuter colleges and universities lack well- 
defined and -structured communities for students 
to establish membership" (p.35). The authors' 
recommendations focus first on building student 
involvement in the classroom through learning 

communities. This is a 
logical emphasis since the 
classroom is the place 
where commuter students 
have the most contact with 
the college. Braxton and 
his co-authors also suggest 
that colleges need to try to 
connect with parents and spouses, since "signifi- 
cant others" have more day-to-day influence on 
commuter students than they do on residential 
students. Braxton and his co-authors also empha- 
size practical considerations such as providing 
courses at convenient times and locations, 
developing jobs on campus (to ease the college/ 
work conflict), and providing day care. 

Their work is an important advance in that it 
explicitly takes account of the special needs of 
non-residential students. Nevertheless, the analysis 
does not differentiate between two- and four-year 
colleges, and the authors suggest that this is 
something that remains to be done. Overall, the 
book leans more toward a four-year college 
perspective, partly because it relies on a body of 
research that is dominated by four-year college 
studies. For example, the book contains no 
discussion of issues related to developmental 



Academic advising should 
be designed to increase 
goal commitment 
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education or to students with explicit occupational 
goals. Community college faculty and administra- 
tors would generally agree that the problems of 
academically underprepared students and the 
developmental programs designed to help those 
students need to be central components of any 
framework for understand- 
ing and improving commu- 
nity college retention. 

What does empirical 
research suggest about these 
models? Although many 
methodological problems 
persist, many studies do 
suggest that academic and 
social integration have 
positive effects on the persistence of four-year 
college students, especially residential students 
(Braxton, Hirschy & McClendon 2004,- Cabrera, 
Nora & Castaneda, 1993). In an extensive review 
of research on college student outcomes, Pascarella 
and Terenzini (1991) found that both the 
frequency and quality of interactions with peers 
and faculty and the participation in extracurricular 
activities — measures of social and academic 
integration into the college life — contribute 
positively to students' persistence at baccalaureate - 
granting institutions. However, the effect is weaker 
when factors such as student characteristics, pre- 
college experiences and other college experiences 
are taken into account. 

Interestingly, Tinto's model (1997, 1993) has 
had more influence on community college research 
than Beans and Metzner's framework (1985), 
despite the latter's attempts to address the 
problems of nontraditional students. Many studies 
of community college attrition are explicitly set up 
as tests of the student integration model, but 
research on social engagement by community 
college students is at best mixed. Pascarella, Smart 
and Ethington (1986) found that academic and 
social integration did have positive effects on 
persistence, but they used a specialized sample, 
which included first-time college students enrolled 
in community colleges in 1971 and who aspired to 



a bachelor's degree. 5 In their review of integration 
research on community college, Bean and Metzner 
(1985) concluded that "social integration is rarely a 
major factor in attrition decisions" at commuter 
institutions (p. 520). In their 1991 review of 
empirical research on undergraduate student 

attainment, Pascarella and 
Terenzini contended that 
these models do not work 
as well for commuter 
colleges as for residential 
colleges, stating that "with 
a few exceptions, the 
weight of evidence is clear 
that various measures of 
social integration (includ- 
ing interaction with faculty, interaction with peers 
and extracurricular involvement) show little if any 
positive relationship with persistence at commuter 
institutions. This lack of a positive relationship 
holds regardless of the specific measure of social 
integration used and irrespective of whether or not 
student background characteristics were taken into 
account in the study design" (p. 402). 

Subsequent research, not reviewed by 
Pascarella and Terenzini, continued to find mixed 
effects. Bers and Smith (1991) and Napoli and 
Wortman (1998) found small positive effects, 
Borglum and Kubala (2000) found no effect, and 
Nora, Attinasi and Matonak (1990) found a 
negative effect of social integration on persistence. 
But all of these results came from single-institution 
studies that determined levels of integration based 
on answers to surveys administered to the students,- 
these studies do not measure the influence of policies 
to promote social integration on persistence. 

In a recent review of empirical tests of Tinto's 
integration model, Braxton, Hirschy and 
McClendon (2004) found strong empirical support 
for the application of the model to residential 
colleges and universities. For commuter universi- 
ties, they found "modest" support for the role of 
social and academic integration in promoting 
"commitment to the institution" if not persistence 
itself (pp. 16-17). With respect to the model's 



Many studies suggest 
that academic and social 
integration have positive 
effects on persistence. 
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usefulness for community colleges, these authors 
stated: "Given this configuration of support [the 
pattern of empirical support that they found in 
their review], the explanatory power of Tinto's 
theory to account for student departure in two- 
year colleges remains undetermined and open to 
empirical treatment" (pp. 17-18). 

What are the implications of these theoretical 
perspectives for institutional practice? In particular, 
the concept of social engagement is probably most 
relevant for a college experience typified by the 
residential liberal arts college with multifaceted 
interactions inside and outside the classroom 
among students and between students and 
professors. Given the nature of their students and 
the large number of part-time faculty, trying to 
reproduce the liberal arts/residential ideal may not 
be the best strategy for community colleges or 
their students. The empirical record is certainly 
consistent with this conclusion. 

With respect to institutional practice, Bean and 
Metzner (1985) emphasize the importance of 
academic advising , presumably to influence students' 
goal commitment and "intent to leave." We review 
the research on advising later in this report. The 
availability of courses, another variable in the Bean 
and Metzner model, has not played an important 
role in subsequent empirical research — perhaps 



because most studies are about a single institution 
— yet this ought to be a promising direction for 
research. Braxton, Hirschy and McClendon (2004) 
also emphasize course availability, and they add 
day care. Other factors that might be particularly 
important for community college students include 
convenient transportation , high-cjuality online 
education , applied pedagogies and well-designed 
internships. These features may be more important 
to community college students than the nature of 
their relationships with professors or participation 
in student organizations. 6 Thus, researchers should 
certainly be looking for other institutional 
variables that do a better job of matching the 
needs of the typical part-time, working community 
college student — needs that are as much practical 
as they are psychological. 

The one place where the engagement model 
may be most relevant at the community college is 
in the classroom. This, after all, is where even 
commuter students interact with faculty and 
potentially with other students. Designing the 
classroom experience to promote more meaningful 
interaction among students and teachers is one 
promising strategy for community colleges,- we will 
examine efforts to do this later, during our 
discussion of learning communities. 



14 PATHS TO PERSISTENCE 





Empirical studies of the effect of 
institutional management and practices 



M uch research in the academic 
literature has been concerned 
with testing various theories of 
student persistence and 
completion. As we have 

emphasized, however, little of this work directly 
tests the effects of institutional practice on student 
outcomes. Another genre of research asks a more 
direct set of questions to determine whether 
students who participate in a particular type of 
program or activity persist or graduate at higher 
rates. In this section we turn to studies that take 
this more straightforward approach. 

Although there are many studies on a long list 
of possible program activities, we focus here on 
four large categories: student services (such as 
advising, counseling, mentoring and orientation 
programs), learning communities, developmental 
education and college-wide reform. Our purpose is 
to draw substantive conclusions and to illustrate 
the status of research on the effectiveness of 
practices at community colleges. 



Advising, counseling, mentoring and 
orientation programs 

Colleges have been experimenting with various 
types of advising and student services for decades. 
According to Pascarella's and Terenzini's summary 
on the effect of such programs, "the most consis- 
tently effective program format appears to be a 
first-semester freshman seminar that meets as a 
regular class with an assigned instructor. The 
purpose of this type of seminar is to orient the 
student to the institution and its programs and to 
teach important academic survival skills" 

(Pascarella and Terenzini, 1991, p. 403). Muraskin 
and Wilner (2004), in their review of institutional 
practices, also concluded that freshman-year 
programs were effective. Nevertheless, both sets of 
authors acknowledged that participation in the 
programs was voluntary. Therefore, the positive 
association might be influenced by initial student 
characteristics and not the service itself. Still, the 
consistency of the findings gives more weight to 
the positive conclusions. 
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Evidence on the effectiveness of other types of 
advising and counseling is more mixed. Pascarella 
and Terenzini (1991) concluded that when 
students' pre-college characteristics were taken 
into account, relatively short-term orientation 
programs had a trivial and statistically insignificant 
direct effect on persistence. They also found mixed 
results from research on the effects on persistence 
of the amount and quality of academic advising. 
Muraskin and Wilner (2004) reported that 
persisting students tended to express higher 
satisfaction with counseling services than did those 
who left, although research on the impact of such 
services was also mixed. The authors also sug- 
gested, though, that 
measurement of counseling 
effectiveness may be 
distorted because students 
who use counseling may 
have come to the college 
with more problems than 
those who do not use it. 

This is a situation in which failing to account for 
differences in the characteristics of program 
participants and non-participants may understate 
the effectiveness of a program. 

Early intervention for academically weak 
community college students, through counseling 
or other student support services, is thought to 
improve their persistence and academic perfor- 
mance (Grubb, 2003; Summers, 2003). Summers 
(2003), in a review of the literature on the impact 
of counseling on attrition, indicated that studies 
have found that counseling increases the retention 
of students who are identified as highly likely to 
drop out. 

Student Support Services (SSS), funded under 
the federal TRIO programs, is perhaps the most 
widespread student services initiative. SSS was 
evaluated in the mid-1990s, and the study found 
that "freshman -year SSS participants increased 
their grade point averages by 0. 15 in the first year 
and 0.11 in the second year of college. SSS 
participation also increased retention to the second 
year of college at the same institution by 7 percent 



and retention to the third year in any institution by 
3 percent" (Muraskin, 1997, p. 1). Interestingly, the 
study also found that effects increased with 
increased exposure to SSS activities. The author 
identified peer tutoring, workshops and cultural 
events as effective components, with peer tutoring 
shown as particularly effective. This was a 
comprehensive study, and the author was careful to 
compare the characteristics of participants to non- 
participants,- nevertheless, he cautioned that 
unmeasured motivation still might have influenced 
both enrollment and program effects. Finally, the 
evaluation did not focus particularly on community 
colleges. Indeed, only one of the five colleges that 

were used in a follow-up 
benchmarking study was a 
community college 
(Muraskin, 1997). 

Thus, the studies on 
counseling and advising 
primarily concern four-year 
colleges. Research on these 
services, and on institutional practices in general, is 
much scarcer for community colleges. For example, 
a recent review of attrition research at community 
colleges by Summers (2003) cited only two 
unpublished single-institution studies on student 
support services, although it reported positive 
effects of "matriculation" services such as assess- 
ment, orientation and counseling. 

The experience of the Community College of 
Denver (CCD) has been particularly influential in 
the discussion of policy and practice in community 
colleges. A book by Roueche, Ely and Roueche 
(2001) provides a description of many of the 
practices and strategies that the college used. 
CCDs counseling and academic support services 
are organized in a comprehensive unit called the 
Academic Support Center (ASC). The college 
reports that the class withdrawal rate was 7.8 
percent for students receiving ASC support, while 
the overall campus rate was 12.4 percent. Although 
this finding is encouraging, the reports do not 
include information that would allow a judgment 
about how students got access to these services 



Studies on counseling and 
advising primarily concern 
four-year colleges. 
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and the comparability of the ASC students with 
other students at the college. 

Research on counseling and student services 
suggests that such programs can play an important 
role in retention and graduation. At the same time, 
this body of research supports two generalizations 
about research on community college practices. 
First, although formal counseling would seem to be 
particularly important for community college 
students (who are likely to have fewer informal 
sources of information and guidance than students 
at four-year institutions), the large majority of 
research on any particular 
higher education practice is 
limited to four-year 
colleges. Second, while 
there are some encouraging 
results from the smaller 
amount of community 
college research, the 
material published from that research does not 
permit a judgment about the validity of its 
conclusions. 

Learning communities 

In the last 15 years, educators in both two- and 
four-year institutions have experimented with 
learning communities as a means of engaging and 
motivating students. Learning communities 
typically organize instruction around themes, and 
students go through such programs as cohorts. 
Learning communities are designed to provide 
more coherent and engaging experiences than 
traditional courses, and to give students and 
faculty more opportunities for increased intellec- 
tual interaction and shared inquiry (Knight, 2002 ; 
Smith et al., 2004,- Tinto, 1997,- Tinto & Love, 
1995). The learning community model is particu- 
larly interesting for community colleges because it 
is one way that these commuter institutions can 
engage with their students in a more intensive way 
than normally occurs in the classroom (Braxton, 
F4irschy & McClendon, 2004,- Fogarty & Dunlap, 
2003; Grubb & Associates, 1999). If the student 



integration model does apply to community 
colleges, it would probably be implemented with 
classroom-oriented approaches such as learning 
communities. In fact, many community colleges 
have adopted various forms of learning communi- 
ties, specifically as a strategy to develop a more 
coherent intellectual environment and forge 
stronger links with the diverse and fragmented 
community college student body (Fogarty & 
Dunlap, 2003,- Smith et al., 2004). 

On the other hand, the design of learning 
communities may discourage nontraditional 

students from participating. 
Because a learning 
community often requires a 
cohort of students to 
attend several classes as a 
group, it may be difficult 
for working students or 
students attending part 

time to participate. Because learning communities 
require close coordination among professors, they 
appear to be most effective when regular, full-time 
faculty members are used rather than part-time 
adjuncts. Yet, full-time faculty are more likely to 
teach during the day, rather than at night when 
many nontraditional students attend classes. These 
factors suggest that learning communities may 
attract more middle-class, traditional-age students 
among those enrolled in community colleges. 7 

In 2003, the National Learning Communities 
Project at The Evergreen State College in 
Olympia, Wash., published an extensive review of 
more than 100 studies of the effectiveness of 
learning communities (Taylor et al., 2003). What is 
remarkable about this review is the length to 
which the authors went to find unpublished as well 
as published research. As a result, this publication 
gives an unusually comprehensive picture of the 
state of research on this topic, including, indeed 
predominantly, studies conducted by institutional 
researchers at colleges. The authors concluded 
that "a preponderance of studies indicate that 
learning communities strengthen student retention 
and academic achievement" (p. iii). 



Learning communities 
typically organize instruction 
around themes. 
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Thus, research on the effects of learning 
communities is encouraging, yet the comprehen- 
sive review of effectiveness studies also illustrates 
many of the challenges to community college 
research (Taylor et al., 2003). For example, the 
large majority of such studies were unpublished, 
single-institution assessments,- only one-seventh 
were published in journals or books. Therefore, 
while there is a tremendous amount of material on 
learning communities, much of it is difficult to 
obtain. Some of the reported studies based their 
assessments on comparisons between learning 
community and non- 
learning community 
students, controlling for 
entering academic charac- 
teristics. But without 
knowing more about the 
process used to recruit and 
enroll students into learning 
communities, it is difficult to judge whether these 
controls were adequate to account for initial 
differences between learning community and 
comparison students. 

Not surprisingly, learning communities in four- 
year colleges have received much more attention 
than those in community colleges,- only 32 of the 
1 19 studies, or about one-quarter, covered 
community colleges. Taylor et al. (2003) chose 17 
studies that were “deemed notable for the quality 
of the assessment study and the manner in which it 
was reported" (p. 4). Of these, only one was about 
a community college program. The review does 
not discuss issues associated with the design and 
effectiveness of learning communities at commu- 
nity colleges — in particular, how effective the 
innovation is for the more typical part-time 
community college student. 

The best-known evaluation of community 
college programs is the 1997 article by Tinto 
published in the Journal of Higher Education , 
“Classrooms as Communities: Exploring the 
Educational Character of Student Persistence," 
which discussed learning communities at Seattle 
Central Community College. Tinto found that 



participation in a learning community did increase 
the probability of quarter-to-quarter persistence. 
Based on a qualitative analysis, he argued that 
learning communities promote persistence by 
facilitating the creation of supportive peer groups 
among students, encouraging shared learning, and 
giving students the opportunity to actively 
participate in knowledge creation. This study used 
a multivariate methodology that controlled for 
possibly confounding characteristics. Tinto also 
recognized the potential distortion in the model 
caused by student self-selection, and he presented 

an argument for explaining 
why it was not a problem. 8 
In contrast to the results 
from the Seattle study, an 
earlier study of learning 
communities at LaGuardia 
Community College by 
Tinto and Love (1995) 
found that participation in learning communities 
did not significantly increase the probability of 
persistence. 

In 2003, the research organization MDRC 
started an evaluation of learning communities at 
Kingsborough Community College. The study 
involves full-time freshmen between ages 17 and 
34, most of whom had applied directly to 
Kingsborough after missing the application 
deadline for the City University of New York 
(CUNY) system. 9 Other students who had low 
placement test scores were also invited to 
participate. Students were assigned, through a 
random selection process, to learning communities 
or to a control group that received services 
generally available to students at the college. The 
course-taking patterns of both groups were tracked 
through the second semester. Preliminary and 
unpublished results suggest that the learning 
community students had passed more courses 
(including basic English classes) and had higher 
grade point averages than the comparison group, 
with differences that were statistically significant. 
There were other encouraging, although not 
statistically significant, differences. 10 In summary, 



Only 32 of the 119 studies, or 
about one-quarter, covered 
community colleges. 



18 PATHS TO PERSISTENCE 




learning communities are an attractive strategy 
with encouraging empirical support. The learning 
community approach also is one strategy that is 
common in community colleges and that is 
consistent with the dominant theoretical perspec- 
tive on student retention. That is, learning 
communities offer the potential for more in-class 
engagement with commuter 
students, who may not have 
a chance to participate in 
social and other extracur- 
ricular activities at the 
college. The use of learning 
communities should be 
expanded while outcomes 
continue to be tracked. In 
the future, researchers need 
to better account for the student recruitment and 
enrollment process in their analyses,- also, they 
should examine the particular problems associated 
with scheduling and formatting learning communi- 
ties so they can be most effective and convenient 
for nontraditional community college students. 

Developmental education and services for 
academically underprepared students 

Improving the effectiveness of developmental 
(remedial) education is perhaps the most important 
issue confronting community colleges today. Most 
entering community college students arrive with 
academic skills that do not allow them to partici- 
pate effectively in at least some college-level 
courses (Perin, forthcoming). Almost one-fifth of 
traditional-aged community college students never 
complete 10 credits, 11 and that number is probably 
higher for older students. Many of these students 
leave because of academic problems and, indeed, 
many students never successfully complete all 
developmental courses that are deemed necessary. 
Thus, as Summers (2003) pointed out: "[M]any 
institutions' primary strategy for reducing attrition 
is the early identification of students likely to drop 
out and the development and implementation of 
intervention services for those students" (p. 64). 



Therefore, colleges offer a variety of services for 
students with weak academic skills. 

Based on their review of the literature on 
academic achievement, Pascarella and Terenzini 
(1991) suggested that institutions can aid the 
academic adjustment of poorly prepared students 
by providing extensive instruction in academic 

skills, advising, counseling 
and comprehensive support 
services. Although they 
reported findings primarily 
from studies of four-year 
colleges, Pascarella and 
Terenzini contended that 
their findings have been 
replicated in several 
national studies and that 
the results hold even after controlling for impor- 
tant student and institutional characteristics. 

Some more recent results are mixed. Many find 
that students who enroll initially in developmental 
courses graduate at lower rates than do students 
who start in regular credit courses (Muraskin & 
Wilner, 2004), although, once again, many of 
these studies analyzed students at four-year 
colleges. In a study of college transcripts, Adelman 
(1998) found that the more remedial courses 
students were required to take, the less likely they 
were to earn a degree. Among students who 
attended two-year and/or four-year institutions and 
earned more than 10 credits, 45 percent of those 
who took two remedial courses earned either an 
associates or bachelors degree by the time they 
were 30 years old, compared with 60 percent of 
those who took no remedial courses. Students who 
are judged to have low reading skills in particular 
are more likely to need extensive remediation and 
less likely to earn a degree. Another study found 
that students at the Community College of Denver 
who started college in developmental classes were 
found to graduate at the same rate as students who 
started in regular courses (Roueche et ah, 2001). 

Researchers studying the effectiveness of 
developmental education face particularly serious 
methodological challenges. On average, students 



Almost one-fifth of 
traditional-aged community 
college students never 
complete 10 credits. 
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who attend developmental education classes start 
out with weaker academic skills. As a result, it is 
hard to identify a causal relationship between 
remedial education and subsequent educational 
attainment. Even if students who start in develop- 
mental classes appear to do more poorly than 
other students, it is still possible that the 
remediation was effective,- without it, the students 
might have done even worse. Comparing similar 
students who do and do not participate in 
developmental education is difficult because many 
states and individual institutions mandate 
remediation for students 
with low assessment scores. 

A study by Bettinger and 
Long (2004) is of interest 
because it uses a statistical 
technique designed to 
identify the causal relation- 
ship in these types of cases, 
although, once again, it is a 
study restricted to four-year 
institutions. The Bettinger- 
Long study focused on a large sample of first-time, 
full-time students of traditional age enrolled in 
Ohio's non-selective public four-year colleges in 
the fall of 1998. The results suggest that students 
placed in remediation are more likely to withdraw 
from college, but they also indicate that participa- 
tion in remedial courses does not seem to decrease 
the likelihood of transferring to more selective 
institutions or attaining a bachelors degree. 

Some studies compare outcomes for different 
types of developmental programs. The best known 
of these types of studies is the National Study of 
Developmental Education, conducted by the 
National Center for Developmental Education 
(Boylan, Bliss & Bonham, 1997). The programmatic 
implications of this and other studies of develop- 
mental education were subsequently published in 
What Works: Research -Based Best Practices in 
Developmental Education (Boylan, 2002). 12 This 
study tested the relative effectiveness of central- 
ized programs (compared with decentralized 
programs), programs with tutorial services with 



trained tutors, programs with advising and 
counseling, and those programs that included 
evaluation on several outcome measures: first-term 
GPA, cumulative GPA, retention in development 
courses and success (earning a D or better) in math 
and English developmental courses. In general, this 
study found more positive results for four-year 
colleges than for two-year colleges, perhaps 
because four-year students may have had stronger 
initial skills . 13 The study did find some positive and 
statistically significant effects for community 
colleges on some of the outcomes listed above. 

The National Study 
(Boylan et al., 1997) 
yielded useful information 
about developmental 
education, and its 
recommendations are in 
accord with the experience 
of many practitioners. 
However, the published 
reports exclude informa- 
tion that would be helpful 
for evaluating the effectiveness of remediation. For 
example, it would be interesting to know the 
magnitude of the effects. Also, because each 
program characteristic is analyzed separately, there 
is no way to determine if a particular program 
component is effective or whether it is effective 
only in combination with other components. 
Finally, it would also be important to know 
whether there are any other institutional features 
such as size, college organization, financial 
condition or typical student characteristics that 
might influence student outcomes and also be 
related to a particular program feature. 

Boylan (2002) lists 33 features that he suggests 
are "best practices." The book references five 
institutions judged to have successful developmen- 
tal programs for benchmarking purposes, but the 
evidence supporting the 33 practices comes 
primarily from other studies and reports, not from 
the five chosen sites. 14 Much of the research in this 
study on developmental/remedial education is 
either based on unpublished documents or material 



It is hard to identify a causal 
relationship between remedial 
education and subsequent 
educational attainment. 
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in the National Center for Developmental 
Educations own publications. Thus, while many of 
these practices are widely used and considered 
effective by practitioners, it is difficult to judge 
whether the evidence supporting these practices is 
definitive. 

Given the pervasiveness of developmental 
education and the controversy surrounding its 
practice, it is surprising that there is still so much 
uncertainty about the most effective approaches to 
working with students with weak academic skills. 
Grubb (2001), in his review of research on 
developmental education, argued that learning 
community formats for developmental education 
did appear to have positive benefits, but he found 
less evidence on the effectiveness of other 
approaches to remedial 
education. To be sure, the 
research is complicated by 
diverse state regulations and 
by local variation in the 
criteria used to place students 
in developmental programs, 
including differing 'cutoff' 
scores on assessment tests. 

Further, many colleges use informal processes to 
alter these regulations on a case-by-case basis 
(Perin, forthcoming). But this overall complexity 
also offers researchers opportunities to make 
comparisons among similar students who do and 
do not enroll in formal remediation or who enroll 
in different types of developmental courses. As 
more detailed transcript-based data become 
available, considerable progress can be made on 
understanding the characteristics and effects of 
these strategies. 

College-wide reform 

So far we have reported on research about 
individual programs. Many practitioners believe 
their programs are successful for the students who 
take part in them. However, although there are 
certainly positive indications for all of these 
practices, we have suggested that an objective look 



at the empirical evidence and the methodologies 
used to test the effectiveness of these programs 
presents a more ambiguous picture. And, though 
many people believe in the effectiveness of 
individual programs, there is much more skepti- 
cism about whether they can be "taken to scale," 
that is, applied as institutional reforms. Comple- 
tion rates at community colleges are low, and 
improving them significantly will probably require 
the successful expansion of pilot programs and the 
strengthening of related programs and services. No 
program, however well designed, can work in 
isolation. An excellent developmental or counsel- 
ing program in a college with generally ineffective 
teaching may ultimately have no effect on student 
completion rates. We have found virtually no 

research that attempts to 
define and assess program 
institutionalization or 
broader college-wide 
reforms. 

There have been 
initiatives designed 
especially to bring about 
reform throughout an 
institution. The best-known initiative for commu- 
nity colleges is the Learning College movement. 
Published work on this model (O'Bannion, 1997) 
presents useful accounts of the processes through 
which colleges have brought about important 
changes, but so far no rigorous assessment of this 
strategy has been published. 

While community college practitioners believe 
that few colleges have been able to bring pilot 
programs to scale successfully, many are convinced 
that the Community College of Denver (CCD) 
has, over the past 20 years, succeeded in bringing 
about fundamental reforms in the basic ways that 
the college operates (Roueche et ak, 2001). These 
reforms followed a systematic planning and 
benchmarking process and have included major 
changes in organization, teaching methods, 
counseling and student services, relationships to 
the community, and organizational philosophy. 

The study of CCD (Roueche et al., 2001) reported 
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increases in the number of graduates and in the 
number of graduates of color and concluded that 
'cohort tracking indicated no significant difference 
in student success on the basis of race, ethnicity, 
age, or gender" (p. 23). This study also reported 
that many CCD students of color transferred 
successfully, that there were high rates of student 
satisfaction with programs and instructors, and 
that surveyed employers demonstrated unanimous 
satisfaction with the skills of CCD graduates. 
These are encouraging results, although, as is true 
for much of the research we have described in this 



analysis, the information reported in the study is 
not adequate to judge whether the measured 
outcomes were caused by the described program 
changes or by other factors. 15 Given the impor- 
tance of the CCD case, further investigation is 
warranted. Careful investigations of other 
institutional change efforts also are important. 
Nevertheless, the CCD experience suggests that 
institution-wide reform can have an effect, and 
that perhaps a focus on individual programs may 
be less effective. 
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Conclusions and implications for practice 



T his report has examined the research 
literature on effective institutional 
practices for improving retention 
and completion at community 
colleges in order to draw substantive 
lessons about those practices and to assess the 
overall state of research in 
the area. We will first review 
the substantive conclusions 
based on our review of the 
empirical research, including 
suggestions for additional 
research. We then present 
conclusions and suggestions 
about the research itself, discussing theoretical, 
data and methodological issues and outlining 
suggestions for strengthening the research that 
takes place at community colleges. 



Substantive questions 

We reviewed the research on the effectiveness 
of four types of practices in increasing persistence 
and completion at community colleges: 1) ad- 
vising, counseling, mentoring and orientation 
programs,- 2) learning communities,- 3) develop- 



Existing research provides the 
most support for learning 
communities. 



mental education and other services for academi- 
cally underprepared students,- and 4) broader, 
institution-wide reform. 

Among the practices and strategies that we 
examined, existing research provides the most 
support for learning communities. The positive 

effects of learning 
communities on persis- 
tence and graduation are 
consistent with the most 
influential theoretical 
perspectives used to study 
retention, and empirical 
research also suggests 

positive effects. Thus learning communities offer 
an approach to connecting more intensively with 
community college students, who often spend 
little time on campus outside of classes. One 
important area for future research involves 
investigation of learning communities for part- 
time, nontraditional students. 

Researchers and practitioners agree that 
counseling and advising are crucial for community 
college students. The national evaluation of the 
federally funded Student Support Services program 
has many useful conclusions, but that study is now 
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a decade old, and the policy and demographic 
environments have changed significantly in the 
intervening years. Many important research 
questions remain to be answered concerning the 
best design of student services, including the 
relationship between counseling and advising and 
the extent to which student services should be 
centralized or distributed among wider groups of 
faculty and staff. 

Developmental education also has received 
considerable attention, and there is widespread 
agreement that it is one of the most important and 
challenging issues facing community colleges. 
Although researchers face difficult methodological 
problems, available data and current empirical 
techniques allow for significant improvement in 
the reliability of conclusions. Moreover, the major 
national study on develop- 
mental education is now 10 
years old. Many issues 
concerning the organization 
and pedagogy of develop- 
mental education warrant 
further investigation. For 
example: Should these 
services be centralized or 
decentralized? Can 
developmental instruction be integrated into 
regular classes? Under what circumstances should 
developmental studies be considered prerequisites 
to other classes? How effective are alternatives to 
specialized developmental classes such as tutoring 
centers? 

Other questions that need attention include: 
How can programs best be combined? What 
factors promote the institutionalization of 
successful pilot programs? Combining different 
reforms into comprehensive college-wide 
initiatives is an attractive prospect, but we are far 
from any concrete understanding about how such 
programs should be combined. Further, moving a 
successful pilot program to college-wide scale may 
be more difficult than implementing the pilot 
alone. Researchers can make important theoretical 
and empirical contributions in this area by 



developing models for organizational change at 
community colleges, by working out methods to 
measure that change, and by assessing the 
effectiveness of those strategies. 

All of the substantive areas we analyzed 
illustrated weaknesses in community college 
research on program effectiveness. These weak- 
nesses include an overemphasis on research on 
four-year colleges,- lack of appropriate data on 
institutional practices,- methodological problems, 
especially having to do with identifying causal 
relationships,- and inadequate reporting, dissemina- 
tion and discussion of research. We present 
suggestions in each of these areas below: 

Theoretical issues: We have emphasized in 
this report that the dominant theoretical perspec- 
tive on retention, the 
student integration model, 
is most appropriate for 
traditional four-year 
students, particularly those 
living on campus. 

Empirical assessments of 
the model for community 
colleges have been 
inconclusive at best. 
Researchers have begun to focus on applying these 
concepts to 'commuter' universities and colleges, 
but more work still needs to be done to take 
account of the particular characteristics of 
community colleges and their students. Future 
work should pay more attention to the needs of 
part-time, working students,- to convenience and 
accessibility,- to the particular problems of students 
with significant academic deficiencies,- and to the 
focused occupational goals of many community 
college students. 

Data availability: Because insufficient data 
exist on institutional practices, most program- 
effectiveness research is based on single-institution 
samples. While these can be useful, their conclu- 
sions are difficult to generalize because effects may 
be based on particular features of the college being 



Many issues concerning the 
organization and pedagogy 
of developmental education 
warrant further investigation. 
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studied. As much as possible, national-level 
databases, such as those created by the National 
Center for Education Statistics, should include 
programmatic detail to allow research on the 
effectiveness of common practices used to improve 
student success. Collabora- 
tive projects within commu- 
nity college districts and 
states, supplemented with 
increasingly available state - 
or district-level student 
record data, also will be 
helpful in understanding 
student persistence and in 
overcoming the limitations of 
single-campus studies. Even with single-institution 
studies, research that makes use of transcript-based 
longitudinal data can provide many important 
insights. Current proposals to collect unit record 
data at the federal level may provide data that 
would be extremely useful in studying student 
success at community colleges. Concerns about 
student privacy may prevent the collection of 
these data, however. 

Empirical issues: The most difficult empirical 
problem in the assessment of retention practices is 
the attribution of causality. Most of the programs 
that we have examined are voluntary, so compari- 
sons of outcomes for participants and non- 
participants need to be interpreted carefully. Are 
differences in outcomes the result of program 
effects or of initial differences between the two 
groups? 

There are several steps that researchers can 
take to purge the analysis of this distortion. 
Multivariate analysis can control for many 
characteristics. Perhaps the most important is some 
measure of pre-program academic ability: If a 
program enrolls more successful students, then it 
may not be surprising that program participants 
are more successful than comparison students. 
Pascarella and Terenzini (1991) generally discussed 
the strength of such controls in the studies that 
they reviewed, but many studies — especially 



unpublished, single-institution studies — fail to 
take this step, or at least fail to report it. However, 
even if available variables are included in an 
analysis, there may be unmeasured differences 
between the two groups. Students who sign up for 

a program may be more 
motivated than those who 
do not, even if their 
measured characteristics are 
similar. Indeed, this 
unobserved motivation may 
be the reason they sought 
assistance in the first place. 
Alternatively, counselors 
may encourage enrollment 
for students whom they judge might "benefit" from 
a program. 

What can researchers do to address this 
problem of comparison group differences? 
Fundamentally, the solutions require an analysis or 
manipulation of the process through which 
students enter a program. One approach is for 
program organizers to enroll a group chosen at 
random from a pool of applicants, thereby 
eliminating any systematic differences between the 
two groups of applicants. 16 Unfortunately, this 
strategy is expensive. For practical reasons, such 
studies cannot usually form the basis of an 
expanded and invigorated research function on 
most campuses. 

A second approach involves conducting a 
statistical analysis of the selection process and 
using those results to adjust the measurement of 
the program effect (see, for example, Bettinger & 
Long, 2004). This is an increasingly popular 
approach in program evaluation, but for the most 
part, it has simply not been used in the community 
college literature. The disadvantage is that this 
approach requires particular types of data that are 
often unavailable and a fairly high level of 
statistical skill. In addition, the results are 
sometimes difficult to interpret. 

When these approaches are not possible, in 
addition to controlling for relevant measurable 
characteristics such as academic achievement, 



The most difficult empirical 
problem in the assessment of 
retention practices is the 
attribution of causality. 
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researchers must include a detailed description of 
the process through which students entered a 
program so that the reader can make a judgment 
about the potential effect that the recruitment and 
enrollment process might have had on the analysis. 
If the selection process tends to enroll more 
motivated students, then the measured program 
effect may be exaggerated. Conversely, the 
enrollment process might result in an understate- 
ment of the program effect. For example, if 
counselors consciously send students with the 
most serious problems to a program, and if 
students in that program do as well as other 
students with similar 
measured characteristics, 
then it may be reasonable to 
conclude that the program 
was effective. Since many of 
the studies of programs 
involve one or a very small 
number of institutions, there 
is no reason why researchers 
cannot provide this type of 

information, yet few of the studies reviewed here 
did so. 



The nation's community 
college leaders have come to 
agree that colleges must be 
more "data-driven." 



Research at the community college: The 
majority of retention research published in 
mainstream education journals is written by 
university-based academics, and these researchers 
have so far largely overlooked the community 
college sector. Though beyond the scope of this 
report, it is an interesting point that researchers — 
who are usually preoccupied by the experience of 
minority, low-income and other underserved 
students — continue to neglect a sector of higher 
education that enrolls nearly half of all under- 
graduates and even larger percentages of 
underserved students. 

Nevertheless, while it is important that 
university-based researchers pay more attention to 
community colleges, most research on effective 
practices at the colleges will likely be carried out at 
the colleges themselves. This is evident from the 
extensive review published by the National 



Learning Communities Project discussed earlier. 
Most of those studies were single-institution 
projects published by the individual institutions 
themselves. Moreover, during the last several 
years, the nation's community college leaders have 
come to agree that colleges must be more "data- 
driven" and shift from a "culture of anecdote" to a 
"culture of evidence." The research on which this 
report is based supports a major initiative begun in 
2003 called Achieving the Dream: Community 
Colleges Count. That initiative is rooted in the 
principle that colleges should base their practice 
on systematic information and research. This 

implies a rise in the 
prominence of institutional 
research and an increase in 
the extent to which faculty 
and administrators use data 
and research to enhance 
the success of their 
students. The growing 
emphasis on analysis of 
student outcomes at both 
the state and the local levels is one encouraging 
development. Statewide data systems and more 
sophisticated cohort tracking on the campuses 
offer many promising opportunities for useful 
analysis. 

Although, in principle, a shift toward "data- 
driven decision making" is not controversial, there 
is no consensus about exactly what such a shift 
entails or what constitutes "evidence." As we have 
seen, there is already some research on the issues 
of interest here, but in many cases, even in well- 
known and influential reports, methodologies are 
preliminary or are inadequately described. For 
example, samples and variable definitions are not 
well defined. Statements are made about what 
successful colleges do without defining how those 
exemplars were chosen. Reports provide too little 
detail about the programs and, especially, about 
how student participants are recruited and 
selected. Many studies look at several components 
of an overall program — for example, counseling 
and mandatory placement in remediation — one at 
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a time and then report the results for each one. But 
it may be possible that those components tend to 
go together, so it is not clear whether one or both 
elements have the effect, if there is an effect. Of 
course, it is possible that many of the details of the 
methodology are simply omitted from written 
reports in an attempt to make them more readable, 
but the reader has no way of knowing that. 

Another barrier to data-driven decision making 
is simply the lack of time or opportunity to make it 
work. Because workloads are typically heavy at 
community colleges, and because the resources 
devoted to institutional research are often scant, 
faculty and administrators have few opportunities 
to develop a deep understanding of outcome data 
and forge a common plan to improve outcomes. 
Indeed, for many community college faculty 
members and administrators, the best opportuni- 
ties to address these issues occur in brief panel 
discussions at conferences — such as those held by 
the American Association of Community Colleges 
or the League for Innovation in the Community 
College — or at one-time faculty workshops. 
Unfortunately, these events 
typically feature researchers 
asserting conclusions, not a 
thorough discussion of 
methods. Definitive and 
dubious results all are 
presented in a similar way, so 
participants can come away 
with a distorted view of the 
reliability of research 
findings. 

Moreover, there is no widespread agreement 
about what constitutes useful analysis. In giving 
colleges advice about how to conduct evaluations 
on developmental education, one influential expert 
stated: "Evaluation of developmental education 
does not require the use of complicated statistics. 
Program outcomes can be accurately described 
using nothing more than percentages, bar graphs, 
and pie charts" (Boylan, 2002, p. 42). These types 
of descriptive statistics are useful to begin 
discussion and investigation, and they are 



important to provide a basis for wide engagement 
of faculty and administrators in discussions of 
policies and practices. Presenting research findings 
in a way that is understandable to a broad audience 
is key to their effectiveness. Nevertheless, 
evaluating the effectiveness of educational 
programs is difficult, and simple descriptive 
information can't paint a full picture. Such data 
should be seen as only a first step in understanding 
program effects. 

In summary, developing a "culture of evidence" 
in community colleges involves a commitment to 
carry out thoughtful research — which often must 
be complex — and an ability to engage faculty, 
administrators and even students in meaningful 
discussions about the implications of that research. 
We present six suggestions for developing that 
culture: 

1 . Colleges must devote more resources and 
skills to research. Some community 
colleges have sophisticated research 
departments with well-trained personnel, 

but many institutional 
research depart- ments 
are staffed by part-time 
researchers who have 
other primary responsibili- 
ties. A college's emphasis 
on institutional research 
demonstrates the depth of 
its leaders' commitment to 
management based on 
information and research. 
While colleges certainly need researchers 
with quantitative skills, qualitative research 
based on interviews, focus groups and 
observations also are important to 
understand and interpret quantitative 
findings. 

2. Colleges must recognize that assessing the 
effectiveness of practices is difficult and 
involves a continuum of activities and 
analyses that range from simple descriptive 



Administrators have few 
opportunities to develop a 
deep understanding of 
outcome data. 
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comparisons to more time-consuming and 
expensive controlled analyses and 
experiments. Randomized experiments are 
considered the 'gold standard/' but they 
are costly and difficult to implement. Short 
of the most ambitious methodologies, 
colleges can make a 
great deal of 
progress by 
employing thought- 
fully designed 
approaches that use 
the increasingly 
available data at the 
campus and state 
levels. 

3. Projects should combine quantitative 
research on student outcomes with 
qualitative research to elicit insights from 
students about those outcomes. Student 
perspectives are crucial for interpreting 
quantitative findings. Analysis of student 
longitudinal data may show that many 
students never complete developmental 
math classes, for example, but focus groups 
and interviews with those students can 
provide insights into why that happens and 
what steps might be successful in improv- 
ing the class completion rates. 



disseminate useful research findings from 
state and institutional research offices. 
State- and college-level researchers 
frequently analyze and assess policies and 
practices, but the results of this work are 
seldom published in ways that are useful to 
wider audiences. Many 
reports of "best practices" 
fail to provide enough 
backup information and 
data for readers to make an 
informed judgment about 
the practices' relative 
effectiveness. Assessment 
results are often only 
available in PowerPoint 
slides, institutional reports, various types of 
testimony, or in the files of institutional 
researchers. 

6. Collaboration among academic, institu- 
tional and state-level researchers should be 
promoted. Researchers working to improve 
the performance of community colleges 
face formidable problems. They will have 
more chance of success if they use a variety 
of methodologies and if they combine 
research based on national, state and local 
data sets, as well as specific institutional 
and state-level knowledge. 



Educators, policy-makers and 
students must move forward 
based on the best 
information available. 



4. Colleges, states and college associations 
must provide more opportunities for faculty 
and administrators to discuss evidence 
about student outcomes. The typical short 
presentations with a few minutes of "Q&A" 
give a distorted view of research results. 
Moreover, thorough college -wide reform 
based on evidence requires broad-based 
participation. Research cannot be the 
province of a few specialists who report 
only to the top administration and board. 

5. Colleges and states must develop more 
systematic methods to publicize and 



We have criticized the quantity and quality of 
research on the effects of institutional practices on 
community college retention and completion rates, 
but we also are pragmatists. We recognize that 
community colleges are ongoing operations and 
that program improvement cannot await definitive 
research results. Educators, policy-makers and 
students must move forward based on the best 
information available, even if that information is 
subject to alternative interpretations. Still, as 
policy-makers, private funders and the public have 
increasingly turned their attention to community 
colleges, we have the opportunity to strengthen 
the available research and provide more useful 
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information as administrators and faculty members 
try to improve students' educational outcomes. 

We suggest that, in planning activities, colleges 
search for the best information they can find. And 
they should search critically, recognizing that all 
research is not the same and that even the most 
definitive studies, such as those using random- 
assignment methodologies, have limitations. At the 
same time, they should do what they can to 
monitor progress and do so as thoroughly and 



rigorously as possible. The interaction between 
research and practice should not be seen as a 
search by experts for the final and definitive 
answer to the question "What works?" Rather, it is 
a constant and continuous process — a conversa- 
tion within and among the colleges and with out- 
side researchers and policy-makers, using the best 
possible data and the most appropriate methodolo- 
gies, as practitioners try to improve their practice 
in a constantly changing environment. 
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Endnotes 



1 . Completion rates in this paragraph are based on 
the authors' calculations from the Beginning 
Postsecondary Student Longitudinal Study of 
1996-2001 (BPS96). 

2. For more information on Achieving the Dream, 
see www.achievingthedream.org . The nine partner 
organizations are: American Association of 
Community Colleges; Community College 
Leadership Program, University of Texas-Austin,- 
Community College Research Center, Teachers 
College, Columbia University,- The Futures 
Project, Brown University,- Jobs for the Future,- 
Lumina Foundation for Education,- MDC Inc.,- 
MDRC,- Public Agenda. 

3. The mainstream journals reviewed were The 
Journal of Higher Education, Research in Higher 
Education, The Review of Higher Education , Journal of 
College Student Development and the NASPA 
Journal. Of these journals, the Journal of Higher 
Education had the highest community college share 
with 1 3 percent. 

4. Authors' calculations from BPS96. 

5. Fifty- three percent completed a bachelor's 
degree within nine years. This baccalaureate 



completion rate is much higher than typical 
contemporary cohorts. 

6. Some of the for-profit institutions emphasize 
convenience over engagement. For example, the 
University of Phoenix has been known for using 
locations and schedules to facilitate the participa- 
tion of working adults and has chosen not to 
promote student involvement by developing 
extracurricular campus activities. 

7. We thank Kate Shaw for this point. 

8. Although Tinto addressed the selection issue, he 
did not provide enough information to make a 
judgment about the extent to which it is a 
problem. 

9. Thomas Brock, the principal investigator, 
reported that Kingsborough institutional research 
indicated that these types of students are often the 
least prepared and have low retention rates. 

10. Personal communication with Thomas Brock, 
the principal investigator of the MDRC study. This 
study is scheduled to be published in early 2005 
after results from a second cohort of Kingsborough 
students become available. 
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1 1 . Authors' calculation from the National 
Education Longitudinal Study (NELS). 

12. Since this study only examined students placed 
in developmental programs or courses, it could not 
measure the effect of developmental programs 
compared with placement into mainstream 
courses. 

13. This type of information is not provided by 
this study. 

14. The study itself presents no evidence of the 
effectiveness of the developmental programs at the 
five chosen sites. Indeed, the sites were chosen not 
on the basis of their outcomes (which are not 
reported), but on the basis of the programs that 
they used. 



15. For example, the levels of satisfaction are 
reported for one point in time, but the reader does 
not know what those rates were before the 
introduction of the college-wide reforms. 
Enrollments and graduation rates improved, but we 
also know that recruitment policies at the college 
changed, resulting in a larger number of tradi- 
tional-aged students,- thus, the changing character- 
istics of the student body may have affected some 
of the results. Additionally, the study did not 
indicate how "student success" was defined and 
what control variables were used in the cohort 
analysis. 

16. The MDRC study discussed earlier takes this 
approach. 
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